A common task in the manipulation of digital images is to extract one or more foreground objects from a background scene. There are a number of difficulties associated with extracting the object in an accurate way, especially when the background is complex and contains several different colors.
One difficulty is that the object can blend in with the background scene, i.e. a pixel at the edge of an object can have contributions from both the foreground and background and the color of the pixel then becomes a blend of the foreground and background colors. Another difficulty relates to object complexity, i.e. an object border may contain substantial detail that requires tedious effort to identify. Furthermore, there may be combinations of these problems, which for example is the case when the object has hair or fur. The hair or fur shapes are complex, and regions with thin fibers lead to substantial color blending. This makes it difficult to correctly extract the object from the background, since it both has to be masked (i.e. it has to be determined which pixels in the border region belong to the object) and it has to be color decontaminated (i.e. pixels having colors from both the object and the background need to be “cleaned up” so they only contain colors of the object).
Several conventional systems and methods addressing these difficulties have been developed, but all of them have various problems as will be discussed below.
One conventional method for extracting an object from a background in a still image is to use a software product, such as Adobe® Photoshop® 5.5 and 6.0. Photoshop provides an image extract feature that enables highlighting of a fuzzy border of an object in a still image. Photoshop performs the image extraction, producing both an opacity mask and modified colors which remove the background contamination from the object. This method has been described in “METHOD FOR DETERMINING A BORDER IN A COMPLEX SCENE WITH APPLICATIONS TO IMAGE MASKING,” application Ser. No. 09/566,632, filed May 8, 2000, which is incorporated herein by reference. The image extract feature can also be applied to a video clip consisting of a series of digital images. However, the image extract feature requires a manual process of highlighting and extracting the object in each frame in the video clip.
Conventional systems that perform image extraction in digital video use a background having a constant, or almost constant, background color (usually blue). Despite the techniques that have been developed in this area, problems still remain with extracting objects whose edges contain substantial detail or have contributions from both the background and the foreground. Blue spill or blue flare (reflection of blue light from the blue screen on the foreground object), backing shadows on the blue screen (shadows of the foreground object on the backing that one wishes to preserve as part of the foreground object), and backing impurities (departures of a supposedly pure blue backing from pure blue) are examples of problems. An analytical approach to overcome some of these problems was presented by A. Ray Smith and J. F. Blinn in “Blue Screen Matting” (Computer Graphics Proceedings, Annual Conference Series, 1996, 259–268) [hereinafter Smith]. Smith proposed a “triangulation solution” where, in addition to a shot of the object against a blue background, another shot of the object is taken against a background having a second color. However, this method cannot be used for moving foreground objects because the method performs a pixel-by-pixel comparison, and the object therefore needs to be in the exact same position in the two shots. Consequently, even though the proposed method may reduce the problems associated with blue screens, it cannot be used when there are moving objects present in a scene with a more realistic background.
It would thus be desirable to have a method similar to the image extract feature in Adobe® Photoshop® that permits accurate object extractions from non-uniform backgrounds, and that also can be applied to a digital video clip without manually having to mark the object to be extracted in every digital image in the video sequence.